1. Field of the Invention
The present invention generally relates to computer systems, and more particularly to an improved method of handling communications between computer components such as processing units of a multi-processor system which are interconnected in a distributed topology.
2. Description of the Related Art
The basic structure of a conventional symmetric multi-processor computer system 10 is shown in FIG. 1. Computer system 10 has one or more processing units arranged in one or more processor groups; in the depicted system, there are four processing units 12a, 12b, 12c and 12d in processor group 14. The processing units communicate with other components of system 10 via a system or fabric bus 16. Fabric bus 16 is connected to a system memory 20, and various peripheral devices 22. Service processors 18a, 18b are connected to processing units 12 via a JTAG interface or other external service port. A processor bridge 24 can optionally be used to interconnect additional processor groups. System 10 may also include firmware (not shown) which stores the system's basic input/output logic, and seeks out and loads an operating system from one of the peripherals whenever the computer system is first turned on (booted).
System memory 20 (random access memory or RAM) stores program instructions and operand data used by the processing units, in a volatile (temporary) state. Peripherals 22 may be connected to fabric bus 16 via, e.g., a peripheral component interconnect (PCI) local bus using a PCI host bridge. A PCI bridge provides a low latency path through which processing units 12a, 12b, 12c and 12d may access PCI devices mapped anywhere within bus memory or I/O address spaces. PCI host bridge 22 also provides a high bandwidth path to allow the PCI devices to access RAM 20. Such PCI devices may include a network adapter, a small computer system interface (SCSI) adapter providing interconnection to a permanent storage device (i.e., a hard disk), and an expansion bus bridge such as an industry standard architecture (ISA) expansion bus for connection to input/output (I/O) devices including a keyboard, a graphics adapter connected to a display device, and a graphical pointing device (mouse) for use with the display device.
In a symmetric multi-processor (SMP) computer, all of the processing units 12a, 12b, 12c and 12d are generally identical, that is, they all use a common set or subset of instructions and protocols to operate, and generally have the same architecture. As shown with processing unit 12a, each processing unit may include one or more processor cores 26a, 26b which carry out program instructions in order to operate the computer. An exemplary processor core includes the PowerPC™ processor marketed by International Business Machines Corp. which comprises a single integrated circuit superscalar microprocessor having various execution units, registers, buffers, memories, and other functional units, which are all formed by integrated circuitry. The processor cores may operate according to reduced instruction set computing (RISC) techniques, and may employ both pipelining and out-of-order execution of instructions to further improve the performance of the superscalar architecture.
Each processor core 26a, 26b includes an on-board (L1) cache (actually, separate instruction cache and data caches) implemented using high speed memory devices. Caches are commonly used to temporarily store values that might be repeatedly accessed by a processor, in order to speed up processing by avoiding the longer step of loading the values from system memory 20. A processing unit can include another cache, i.e., a second level (L2) cache 28 which, along with a memory controller 30, supports both of the L1 caches that are respectively part of cores 26a and 26b. Additional cache levels may be provided, such as an L3 cache 32 which is accessible via fabric bus 16. Each cache level, from highest (L1) to lowest (L3) can successively store more information, but at a longer access penalty. For example, the on-board L1 caches in the processor cores might have a storage capacity of 128 kilobytes of memory, L2 cache 28 might have a storage capacity of 512 kilobytes, and L3 cache 32 might have a storage capacity of 2 megabytes. To facilitate repair/replacement of defective processing unit components, each processing unit 12a, 12b, 12c, 12d may be constructed in the form of a replaceable circuit board or similar field replaceable unit (FRU), which can be easily swapped installed in or swapped out of system 10 in a modular fashion.
As multi-processor, or multi-chip, computer systems increase in size and complexity, an excess amount of time can be consumed by the overall system in performing various supervisory operations, e.g., initializing each chip at boot time (IPL) or for some other system reset. Most of the supervisory commands that are issued from the service processor to each chip are the same, introducing a degree of redundancy in the procedures that causes a small problem in small systems, but scales to a bigger problem as the system gets bigger. An exemplary state-of-the-art multi-processor system might have four drawers of processing units, with two multi-chip modules (MCMs) in each drawer, and four processing units in each MCM, for a total of 32 processing units. This construction leads to a long boot time as the service processor must sequentially send initialization commands to each of the 32 processing units. The problem can additionally arise with other commands that might be issued after initialization, such as cumulative status checking, or reading fault isolation registers (FIRs).
This problem applies to supervisory routines running on the service processor and also any supervisory routines that might be running on one of the processor cores, since a core cannot directly control other chips in the system without communicating with the service processor, which creates a communications bottleneck. Moreover, this type of usage of the service processor represents a somewhat centralized control structure, and the trend in modern computing is to move away from such centralized control since it presents a single failure point that can cause a system-wide shutdown.
In some prior art multi-processor topologies, data pathways may be provided directly between processing units to allow sharing of memory, but these pathways are inappropriate for handling system-wide commands. The inter-chip data pathways have limited functionality, and are part of the clock-controlled domains of the chips. Accordingly, any attempted use of these pathways for supervisory commands would interrupt operation of the processing units and adversely affect overall system performance.
In light of the foregoing, it would be desirable to devise a communications mechanism for a multi-processor computer system which facilitates transmission of system-level (e.g., supervisory) commands to different chip components such as processor cores and memory subsystems. It would be further advantageous if the mechanism could allow such commands to issue and execute while the processing units are running, that is, without interruption.